[BUG] Notion API Error: Unsupported block type ai_block

Hello LlamaIndex team,

I am using the LlamaIndex library to interact with the Notion API and I have encountered an issue. When I try to load pages from Notion using the NotionReader class, I receive an error message stating that the ai_block type is not supported by the API.

Here is the error message I receive:

Error: Notion API error: block type `ai_block` is not supported via the API.

Here is the relevant code from my route.ts file:

import { Message, StreamingTextResponse } from 'ai';
import { OpenAI, NotionReader } from 'llamaindex';
import { NextRequest, NextResponse } from 'next/server';
import { createChatEngine } from './engine';
import { LlamaIndexStream } from './llamaindex-stream';
import { Client } from '@notionhq/client';

export const runtime: string = 'nodejs';
export const dynamic: string = 'force-dynamic';

type Page = {
  metadata: {
    id: string;
    title: string;
    createdTime: string;
    lastEditedTime: string;
    parentId?: string;
  };
  properties: string[];
  lines: string[];
};

type Pages = Record<string, Page>;

export async function POST(request: NextRequest) {
  try {
    const body = await request.json();
    const { messages }: { messages: Message[] } = body;
    const lastMessage = messages.pop();
    if (!messages || !lastMessage || lastMessage.role !== 'user') {
      return NextResponse.json(
        {
          error:
            'messages are required in the request body and the last message must be from the user',
        },
        { status: 400 }
      );
    }

    const pageLoader = new NotionReader({
      client: new Client({
        auth: process.env.NOTION_INTEGRATION_TOKEN,
      }),
    });

    const pageDocs = await pageLoader.loadPages('rootPageId');

    const { llm, relevantInfo } = await extractRelevantInfo(
      pageDocs,
      lastMessage.content
    );

    const chatEngine = await createChatEngine(llm);

    const responseText = generateResponse(relevantInfo);

    const response = await chatEngine.chat(responseText, messages, true);

    const stream = LlamaIndexStream(response);

    return new StreamingTextResponse(stream);
  } catch (error) {
    console.error('[LlamaIndex]', error);
    return NextResponse.json(
      {
        error: (error as Error).message,
      },
      {
        status: 500,
      }
    );
  }
}

async function extractRelevantInfo(
  pageDocs: Pages,
  query: string
): Promise<{
  relevantInfo: string;
  llm: OpenAI;
}> {
  const openai = new OpenAI({
    apiKey: process.env.OPENAI_API_KEY,
    model: 'gpt-4-1106-preview',
    temperature: 0.7,
    maxTokens: 800,
  });

  let combinedContent = '';
  for (const pageId in pageDocs) {
    const page = pageDocs[pageId];
    if (page && page.lines) {
      combinedContent += page.lines.join('\n') + '\n\n';
    }
  }

  const prompt = `This AI extracts information based on the following content. It cannot answer questions about information that is not related to the Notion page.\n\nContent:\n${combinedContent}\n\nQuestion: ${query}\n\nRelevant information:`;
  const response = await openai.complete(prompt);

  return {
    relevantInfo: response.message.content.trim(),
    llm: openai,
  };
}

function generateResponse(relevantInfo: string): string {
  if (relevantInfo) {
    return relevantInfo;
  } else {
    return 'Sorry, I could not find any relevant information.';
  }
}

I understand that this error is due to the Notion API not supporting the ai_block type. However, I am unable to find a way to exclude this block type from the API request or to filter it out after the data is received.

I would appreciate it if you could provide a solution or workaround for this issue. It would be helpful if there was a way to exclude certain block types when using the NotionReader class, or if the class could automatically handle unsupported block types.

Thank you for your assistance.